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AN  ARCHITECTURE  FOR  A  PROBLEM-SOLVING  ASSESSMENT 
AUTHORING  AND  DELIVERY  SYSTEM 

Gregory  K.  W.  K.  Chung,  Eva  L.  Baker,  Girlie  C.  Delacruz,  Jesse  J.  Elmore, 
William  L.  Bewley,  and  Bruce  Seely 
CRESST/University  of  California,  Los  Angeles 


Abstract 

This  report  describes  the  design  of  an  authoring  system  to  support  the  design 
of  problem-solving  assessments.  A  key  component  underlying  the  system 
architecture  is  a  constraint  network.  In  a  constraint  network,  nodes  are  variables  that 
can  assume  a  range  of  values  and  the  topology  specifies  how  the  variables  and 
values  are  related  (Montanari,  1974).  To  support  assessment  design,  the  system 
design  includes  a  constraint  network  describing  the  permissible  relations  and  states 
among  assessment  and  problem-solving  variables. 
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The  UCLA  National  Center  for  Research  on  Evaluation,  Standards,  and  Student 
Testing  (CRESST)  is  under  contract  to  the  Office  of  Naval  Research  (ONR)  to 
conduct  research  on  assessment  models  and  tools  designed  to  support  Navy  and 
Marine  Corps  distance  learning  (DL).  The  project  is  called  Knowledge,  Models,  and 
Tools  to  Improve  the  Effectiveness  of  Naval  Distance  Learning,  or  KMT.  The 
approach  to  conducting  KMT  research  has  been  to  develop  and  test  tools  designed 
to  address  the  assessment  and  training  requirements  posed  by  real  Navy  and 
Marine  Corps  training  applications. 

The  Navy  is  currently  undergoing  a  revolution  in  training.  Future  training,  as 
envisioned  by  Admiral  Clark,  "will  apply  information-age  methods  to  accelerate 
learning  and  improve  proficiency"  (Clark,  2003).  These  methods  include  advanced 
trainers  and  simulators,  tailored-skills  training  programs,  and  more  effective 
performance  measurement  tools.  Distributed  learning  is  a  fundamental  element  of 
the  envisioned  information-age  methods,  in  part  because  of  its  promise  to  provide 
efficient  and  effective  learning  on  demand,  and  in  part  because  of  its  promise  to 
reduce  the  time  Sailors  and  Marines  spend  away  from  their  command  attending 
schools.  The  Navy  is  taking  aggressive  steps  to  support  distance  learning  and  the 
Revolution  in  Training  by  developing  an  Integrated  Learning  Environment  (ILE), 
the  objectives  of  which  include  reducing  content  development  cost  and  lead  time 
through  reuse  of  learning  objects  and  improving  content  relevance  by  reducing  cycle 
times  to  review  and  validate  requirements  (NPDC,  2003). 

To  deliver  these  capabilities,  the  new  Navy  training  organizations  and  training 
development  contractors  need  research-based  human  performance  and  assessment 
knowledge,  models,  and  tools  to  support  the  development  and  use  of  information- 
age  methods.  These  include  guidelines  for  courseware  development  and  evaluation 
and  courseware-authoring  capabilities  including  models  and  tools  for  assessing 
learner  performance  based  on  pedagogically  sound  principles  consistent  with  the 
best  available  knowledge  from  learning  and  assessment  research  and  instructional 
design. 


Authoring  Systems  for  Assessment 

In  his  reflection  on  the  current  state  of  testing,  van  der  Linden  (2005)  observed. 

Any  outsider  entering  the  testing  industry  would  expect  to  find  a  spin-off  in  the  form  of 
a  well-developed  technology  that  enables  us  to  engineer  tests  rigorously  to  our 
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specifications. ...  To  draw  a  parallel  with  the  natural  sciences,  it  seems  as  if  testing  has  led 
to  the  development  of  a  new  science,  but  the  spin-off  in  the  form  of  a  technology  for 
engineering  the  test  has  not  yet  been  realized,  (p.  xi) 

van  der  Linden's  (2005)  observation  highlights  the  craft  nature  of  assessment 
development:  the  lack  of  a  common  knowledge  will  prevent  the  field  from  moving 
from  craft  knowledge  toward  an  engineering  model  for  test  development.  What  is 
needed  is  a  method  to  explicitly  represent  the  assessment  design,  from  which  actual 
assessments  can  be  derived.  Establishing  this  assessment  model,  in  computational 
form,  will  provide  traceability  between  the  assessment  (instantiated  in  a  particular 
content  area  for  a  particular  purpose  and  population)  and  the  assessment  model. 
Thus,  the  development  of  assessments  becomes  rational — particular  features  (good 
or  bad)  of  the  assessment  are  based  on  an  underlying  model.  Because  the  model  is  in 
computational  form,  it  is  persistent  until  modified.  This  transparency  is  a 
prerequisite  for  demystifying  the  test  development  process  and  is  central  to  moving 
toward  a  "technology  for  engineering  the  test." 

One  of  the  most  important  capabilities  of  an  assessment  authoring  system 
would  be  the  shortening  in  the  time  required  to  gather  validity  evidence  for 
different  purposes  (Baker,  2002a;  O'Neil  &  Baker,  1997).  Historically,  the 
development  lifecycle  of  assessments,  particularly  for  measuring  complex  learning 
(e.g.,  performance  assessments),  requires  significant  amounts  of  time  and  resources 
(O'Neil  &  Baker,  1997).  Thus,  an  authoring  system  should  have  the  capability  to 
rapidly  generate  tasks  appropriate  for  different  assessment  purposes. 

When  authoring  is  applied  to  the  field  of  testing,  additional  requirements  come 
into  play.  With  assessment  and  testing,  the  key  requirement  is  validity,  that  is,  the 
extent  to  which  inferences  drawn  from  the  result  of  the  test  or  assessment  are 
warranted  (Messick,  1995).  Linn,  Baker,  and  Dunbar  (1991)  have  described  essential 
elements  of  validity  applied  to  open-ended  assessment  tasks.  These  validity  criteria 
include  cognitive  complexity,  linguistic  appropriateness,  transfer  and 
generalizability,  content  quality,  reliability,  and  instructional  sensitivity.  Moreover, 
when  designing  an  authoring  system  (rather  than  a  test,  for  example)  one  is 
interested  in  the  utility  of  the  system  for  its  users  (teachers  or  test  developers)  in 
addition  to  the  value  of  the  data  yielded  by  administering  tests  to  students. 

In  testing,  it  is  often  the  case  that  instructors  who  need  to  use  tests  and 
assessments  routinely  in  their  classrooms  have  little  time  and  expertise  to  create 
high-quality  assessments  of  student  learning.  They  may  use  a  craft  approach. 
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creating  each  test,  one  at  a  time,  with  a  wholly  new  format,  scoring  approach,  and 
set  of  cognitive  requirements.  This  approach  generally  produces  tests  of  low  quality 
whose  inferences  may  be  suspect.  As  instructors  attempt  to  bring  all  students  up  to 
high  challenging  standards,  there  is  a  concomitant  desire  to  test  performance  in  such 
a  way  as  to  stimulate  complex  cognitive  processing.  Usually,  instructors  use  essay  or 
other  extended  written  examinations  to  elicit  such  performance.  These  types  of  tests 
are  difficult  to  calibrate,  take  considerable  time  and  cost  to  evaluate,  and  frequently 
result  in  low  reliability  in  scoring.  A  straightforward  way  is  needed  for  instructors  to 
create  assessments  that  require  students  to  demonstrate  complex  knowledge 
representations,  including  declarative,  procedural,  and  systemic  knowledge.  In  this 
report  we  describe  such  an  approach. 

Prior  and  Current  Assessment  Authoring  Systems 

Prior  work  related  to  assessment  authoring  systems  is  anchored  at  three  points. 
At  one  end  are  numerous  resource  Web  sites  that  simply  house  existing 
assessments,  whose  quality  is  unknown.  PALS  (Quellmalz,  Hinojosa,  Hinojosa,  & 
Schank,  2000)  has  been  one  of  the  few  efforts  to  adopt  a  systematic  vetting 
procedure  to  ensure  that  the  assessments  housed  at  their  site  meet  a  set  of  standards 
related  to  validity  and  reliability,  but  even  so,  using  the  assessments  appropriately 
requires  a  relatively  high  level  of  assessment  knowledge.  More  commonly, 
assessment  sites  are  primarily  warehouses  for  tasks  with  undefined  technical 
properties  (e.g.,  Eisenhower  National  Clearinghouse,  2001;  Harvard  Graduate 
School  of  Education,  2001;  Queensland  Association  of  Mathematics  Teachers,  1998). 
Search  and  filter  capabilities  are  usually  provided  to  make  it  possible  to  retrieve 
assessments  by  different  criteria  (e.g.,  grade,  task  type,  domain,  standard).  At  the 
second  point  are  systems  that  provide  the  means  to  build  (but  not  design) 
assessments.  These  systems  typically  exist  within  learning  management  systems  that 
have  the  infrastructure  to  deliver  multiple  test  formats  (e.g.,  multiple  choice,  true- 
false,  short-  and  long-essay).  However,  the  assessments  are  essentially  forms  that 
need  to  be  filled  in  (e.g.,  CAPA,  2000;  Learning  Manager,  2005;  QuestionMark,  2001; 
TRIADS,  2001;  WebCT,  2001).  At  the  third  point  are  authoring  systems  that  are 
targeted  to  assessment  developers  (Chung,  Baker,  &  Cheak,  2001;  Chung,  Klein, 
Herl,  &  Bewley,  2001;  Mislevy,  Steinberg,  Breyer,  &  Almond,  1999;  Osmundson, 
Jeffries,  &  Herman,  1998).  These  systems  build  in  some  underlying  knowledge  about 
valid  assessment  design  (e.g.,  concepts  of  validity  and  reliability)  and  may  require 
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detailed  knowledge  of  the  essential  content.  Other  teacher-focused  systems  such  as 
the  Assessment  Wizard  (Educational  Testing  Service,  2000)  expect  less  assessment 
knowledge,  but  they  assume  substantial  domain  knowledge. 

More  recent  efforts  by  CRESST  and  others  (e.g.,  the  PADI  project)  have 
addressed  the  issue  directly.  For  example,  the  ADDS  system  (Assessment  Design 
and  Delivery  System),  developed  by  CRESST,  is  intended  to  support  teachers' 
creation  of  assessments  mapping  to  standards,  to  challenging  cognitive  demands, 
and  to  subject  matter  content.  Wizard  interfaces  guide  teachers  in  the  specification 
or  selection  of  critical  assessment  criteria,  and  enable  teachers  to  design  their  own 
assessments  by  graphical  composition. 

An  important  rationale  underlying  this  work  is  the  acknowledgment  that  in 
practical  settings,  finding  relevant  science  content  and  examples  for  use  in 
assessment  is  a  time-consuming  bottleneck  for  all  assessment  development.  As  a 
practical  matter,  teachers  have  neither  the  time  nor  expertise  to  find  content  for 
every  assessment.  Performance  assessments  or  other  memorable  extended  tasks  are 
especially  problematic  because  of  potential  learning  effects — that  is,  once  students 
engage  in  an  assessment,  they  may  learn  important  elements  of  the  example  used  in 
the  task  or  even  the  task  itself. 

The  rationale  for  our  approach  is  based  on  more  than  a  decade  of  research  and 
evaluation  on  the  use  of  technology  in  classrooms  (Baker,  2001,  2002b;  Baker, 
Gearhart,  &  Herman,  1994;  Baker  &  Herman,  2000;  Baker,  Herman,  &  Gearhart, 
1996;  Baker  &  O'Neil,  2003;  Chen,  Chung,  Klein,  de  Vries,  &  Burnam,  2000;  Chung, 
Klein,  &  Baker,  2000;  Gearhart,  Herman,  Baker,  Novak,  &  Whittaker,  1994;  O'Neil  & 
Baker,  1993).  We  have  consistently  found  that  teachers'  adoption  of  technology  is 
based  largely  on  the  perceived  utility  to  improve  students'  learning,  and  the  amount 
of  overhead  in  time  and  effort  in  the  non-instructional  uses  (e.g.,  equipment  setup, 
loading  software,  amount  of  technical  support  needed).  Our  design  choices  directly 
address  these  issues  by  minimizing  the  demands  associated  with  technology 
barriers  (e.g.,  standard  Web  access  and  user-friendly  interfaces)  and  knowledge  and 
effort  barriers  (e.g.,  provision  of  default  choices,  automated  access  to  content)  while 
still  providing  the  flexibility  for  teachers  to  design  assessments  for  their  particular 
needs. 
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Research  Questions 

An  implicit  assumption  underlying  the  Navy's  new  training  doctrine  is  the 
availability  of  assessments  capable  of  measuring  and  providing  quality  information 
on  trainees'  knowledge,  skills,  and  abilities.  However,  in  practice,  this  assumption 
may  not  hold  up  for  a  variety  of  practical  reasons  (e.g.,  time  constraints,  not  trained 
in  assessment,  and  so  forth).  Thus,  our  research  effort  has  been  focused  on 
developing  methods  to  support  non-assessment  experts  in  developing  assessments 
consistent  with  good  design  practices.  The  underlying  assumption  is  that  users  will 
not  be  experts  in  designing  assessments,  and  therefore  the  system  should  provide 
support  that  maximizes  the  likelihood  of  the  assessments  conforming  to  modern 
assessment  practices.  Thus,  the  set  of  research  questions  guiding  this  work  are: 

•  To  what  extent  can  a  problem-solving  assessment  framework  be  codified  in 
computational  form?  What  is  an  appropriate  technique  that  is  feasible  and 
transparent? 

•  To  what  extent  can  a  user's  design  of  a  problem-solving  assessment  be 
evaluated  computationally?  This  is  an  important  question  for  practical 
settings  in  which  the  users  are  assumed  to  have  little  assessment  design 
expertise.  Theoretically,  establishing  a  framework  from  which  assessments 
can  be  designed  and  evaluated  against  offers  the  potential  for  coherency, 
generalizability,  and  scalability. 

•  To  what  extent  can  qualitatively  different  assessment  tasks  be  generated 
computationally?  This  question  addresses  issues  of  rapid  deployment  and 
scaling. 


Problem-Solving  Assessment  Framework 

The  assessment  framework  used  in  this  work  is  based  on  Baker  and  O'Neil's 
(2002)  approach  to  designing  problem-solving  assessments  using  computer 
technology.  This  approach  first  characterizes  three  types  of  problem-solving  tasks: 
(a)  a  task  in  which  an  appropriate  solution  is  known  in  advance,  (b)  a  task  in  which 
there  is  no  known  solution  to  the  problem,  and  (c)  a  task  that  requires  an  application 
of  a  given  tool  set  to  a  broadly  ranging  set  of  topics.  Baker  and  O'Neil  highlight  the 
relevant  variables  that  characterize  these  types  of  problems  while  focusing  on 
complex,  scenario-based  problem-solving  tasks.  Table  1  shows  the  variables, 
possible  values,  and  definitions  for  each  variable  and  value. 
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Identifying  the  problem  is  often  one  of  the  most  difficult  aspects  of  problem 
solving  (see  Baker  &  O'Neil,  2002).  The  ambiguity  of  problem  identification  may  be 
dependent  upon  the  prior  knowledge  that  is  required  of  the  examinee,  as  well  as  the 
purpose  of  the  assessment.  An  assessment  author  could  adjust  its  difficulty  by  either 
stating  the  problem  explicitly  or  obscuring  it  in  an  embedded  setting.  Likewise,  the 
information  sources  that  make  up  the  scenario  can  vary  in  quality  from  their 
accuracy  to  their  credibility. 

The  problem  to  be  solved  can  also  be  multiply-masked  in  which  the  solution  to 
one  part  of  the  task  determines  the  nature  of  the  next  part  of  the  task.  An  examinee 
may  be  required  to  mentally  test  various  hypotheses  for  solving  the  problem  under 
extreme  time  constraints  (e.g.,  an  enemy  is  about  to  attack  but  you  are  not  certain 
whether  they  are  armed  with  rifles  or  tanks,  or  a  house  is  on  fire  and  you  are  unsure 
where  the  source  of  the  fire  is)  and  demonstrate  proficiency  in  contingency  planning 
in  order  to  recover  from  an  error  (e.g.,  selected  plan  of  attack  underestimated  the 
enemy's  weapon  power,  or  fire  extinguisher  does  not  work). 

Baker  and  O'Neil  (2002)  emphasize  the  advantage  of  problems  that  are 
sequential  and  conditional  in  nature  both  because  of  their  reflection  of  reality  and 
their  potential  to  measure  competence.  Computerized  assessments  take  advantage 
of  this  ability  to  measure  an  examinee's  proficiency  of  either  the  task  outcome  or  on 
parts  of  the  task.  In  this  environment,  an  examinee  can  execute  an  action  and  either 
be  given  permission  to  continue,  be  given  the  opportunity  to  correct  an  invalid 
procedure,  or  be  provided  with  a  partial  solution  in  order  to  proceed  with  the  rest  of 
the  problem.  By  doing  this,  one  can  get  a  measure  of  an  examinee's  competence  and 
incompetence  on  the  relevant  skills  throughout  the  task.  This  affords  the  instructor 
the  capacity  to  tailor  instruction  to  that  with  which  the  examinee  has  trouble,  rather 
than  to  instruct  on  every  aspect  of  the  task. 
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access.  The  student  will  need  to  look  at  other  sources  to  find  out  that  information. 

Time  constraints  Time  demands  of  the  problem  itself.  Time  constraints  can  assume  two  values:  (a)  external — Something 

(external,  self-paced)  311  the  problem  space  determines  the  time  demand;  and  (b)  self-paced — Completion  of  the  task  is  not 

constrained  by  time. 
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there  is  no  reason  to  question  information  from  this  source;  and  (d)  high  credibility — 1  here  is  never 
any  reason  to  question  information  from  this  source — typically  does  not  require  interpretation  or 
judgment. 
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Use-Case  Scenarios 

In  the  first  part  of  the  concept  of  operations  we  outline  three  scenarios  typical 
of  educational  and  training  settings.  These  scenarios  illustrate  an  instructor's  view  at 
the  beginning,  during,  and  at  the  end  of  a  course  with  respect  to  gathering 
information  about  students.  Following  the  scenarios  we  describe  how  our 
architecture  supports  these  scenarios.  We  then  provide  two  examples  of  the 
authoring  process  in  a  marksmanship  context  (one  sound  assessment  design  and 
one  poorly  designed  assessment). 

Scenario  1:  Determining  what  students  already  know.  The  following  scenario 
is  typical  of  what  instructors  are  confronted  with  at  the  beginning  of  instruction  in 
general.  This  is  a  particular  problem  in  training  settings  where  instructors  are 
confronted  with  waves  of  students  with  whom  the  instructors  are  unfamiliar  and  the 
trainees  have  varying  backgrounds  and  experience. 

A  new  training  course  is  starting  and  the  students  are  coming  with  a  wide  range  of 
backgrounds.  The  instructor  wants  to  get  a  sense  of  what  each  student  knows  about  the 
topic  so  that  instruction  can  be  adjusted  to  fit  with  the  students'  background.  The 
instructor  needs  to  decide  on  what  to  do  given  the  constraints  on  instructional  time, 
effort,  and  anticipated  payoff  for  gathering  information  about  the  knowledge  and  skills 
of  incoming  students. 

Scenario  2:  Determining  how  much  students  are  learning.  The  following 
scenario  is  typical  of  instruction  in  general.  Two  issues  are  central  to  this  scenario: 
how  instructors  determine  what  students  are  learning  and  how  timely  that 
information  is  (i.e.,  can  instructors  act  on  the  information).  In  practice,  these  issues 
are  conditioned  by  the  real-world  constraints  such  as  the  amount  of  instructional 
time,  the  skill  of  the  instructor  in  terms  of  instructional  and  assessment  skill,  and  the 
capacity  available  (e.g.,  whether  the  instructor  has  available  time,  or  an  assistant  to 
score  tests). 

During  the  duration  of  the  training  course  the  instructor  wants  to  get  a  sense  of  what 
each  student  is  learning  so  that  instruction  can  be  adjusted  to  maximize  student  learning. 

In  an  ideal  instructional  situation,  the  instructor  has  information  on  how  well  each 
student  is  comprehending  the  content,  how  the  student  applies  the  knowledge  to  novel 
situations,  what  topics  students  find  confusing,  what  topics  are  redundant,  and  what 
topics  need  more  elaboration.  Further,  this  information  is  accurate  and  at  such  a  grain 
size  that  the  instructor  can  take  immediate  action  to  adjust  instruction  and  content.  The 
instructor  needs  to  decide  on  what  to  do  given  the  constraints  on  instructional  time. 
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effort,  and  anticipated  payoff  for  gathering  information  about  the  knowledge  and  skills 
of  current  students. 

Scenario  3:  Determining  the  degree  to  which  students  have  learned  requisite 
skills  and  knowledge.  An  important  question  at  the  end  of  the  course  is  whether 
students  have  acquired  the  knowledge  and  skills  intended  by  instruction. 

Near  the  end  of  the  training  course  the  instructor  wants  to  determine  whether  students 
have  learned  the  content  and  skills  specified  in  the  course  objectives.  In  an  ideal 
instructional  situation,  the  instructor  has  had  ongoing  information  about  the  overall 
condition  of  the  class  and  thus  the  instructor  should  have  a  reasonably  accurate  picture 
of  whether  the  course  objectives  were  met.  Thus,  any  testing  at  the  end  of  the  course  is 
verification  that  students  have  attained  a  particular  level  of  skill  and  knowledge.  Passing 
the  test  should  reflect  that  the  student  has  met  the  standard  set  for  the  course.  The 
instructor  needs  to  decide  on  whether  to  administer  a  test  and  if  so,  what  students  have 
to  do  to  demonstrate  content  and  skill  mastery.  Practical  constraints  include  instructional 
time,  effort,  anticipated  payoff,  and  whether  the  test  is  a  good  certification  for  the  course. 

In  each  of  these  scenarios,  the  typical  set  of  options  include  the  following 
(ordered  by  amount  of  in-class  time): 

1.  Do  nothing,  assume  students  are  learning 

2.  Whole-group  questioning  of  students  (show  of  hands) 

3.  Whole-group  testing  (short) 

4.  Whole-group  testing  (in  depth) 

5.  Individualized  questioning  of  each  student 

6.  Individualized  testing  of  each  student 

With  the  exception  of  the  first  three  options,  there  is  generally  neither  available 
class  time  to  conduct  the  testing,  nor  the  expertise  to  develop  the  test,  nor  the 
capacity  to  score  the  tests.  Further,  in  many  training  and  education  situations,  a 
highly  desired  outcome  is  a  student  who  can  use  the  knowledge  and  skills 
appropriately  in  novel  and  complex  situations — problem  solving  and  transfer. 
However,  practical  constraints  and  limitations  in  expertise  suggest  that  testing 
students  for  problem-solving  skill  and  transfer  will  not  become  common  practice 
without  a  support  structure  that  simultaneously  makes  the  development  and  testing 
process  more  feasible  (i.e.,  fit  within  the  practical  constraints  of  the  classroom)  and 
provides  useful  information  to  the  instructor  that  can  be  acted  on  to  improve 
instruction  immediately. 
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Toward  Authoring  System  Supports  for  Developing  Problem-Solving  Tasks 

A  basic  assumption  underlying  the  architecture  is  that  the  assessment  structure 
be  kept  independent  of  the  domain  structure.  The  purpose  of  this  separation  is 
generalizability  and  scalability.  With  respect  to  generalizability,  an  independent 
assessment  structure  provides  a  substrate  upon  which  all  tasks  are  developed.  A  set 
of  common  design  criteria  increases  the  likelihood  that  the  assessment  tasks  will 
inherit  properties  of  the  assessment  structure.  A  set  of  desirable  outcomes  is  that  the 
system  will  allow  the  generation  of  tasks  that  are  grounded  in  a  theoretical 
framework,  have  a  design  traceable  to  a  constraint  network,  and  increase  the  overall 
quality  of  tasks  by  providing  structure  and  guidance  for  authors  with  little 
knowledge  of  assessment  design.  The  architecture  described  in  this  report  reflects 
our  current  progress  in  developing  such  a  system. 

The  key  technical  challenges  for  developing  a  domain-independent 
representation  for  the  assessment  of  problem  solving  are: 

•  Identifying  the  key  variables  that  represent  the  domain  of  problem  solving 
with  respect  to  assessment. 

•  Identifying  the  set  of  states  the  variables  can  assume. 

•  Codifying  the  relationships  among  variables. 

The  key  technical  challenges  for  developing  a  domain -dependent  representation 

are: 

•  Instantiating,  in  a  particular  domain,  the  domain-dependent  correlates  of 
the  assessment  variables. 

•  Ensuring  adequate  domain  coverage  to  provide  content  and  context  rich 
enough  to  exercise  examinees'  problem-solving  skills. 

An  essential  component  underlying  the  system  architecture  is  a  constraint 
network.  In  a  constraint  network,  nodes  are  variables  that  can  assume  a  range  of 
values  and  the  topology  specifies  how  the  variables  and  values  are  related 
(Montanari,  1974).  To  support  assessment  design,  the  constraint  network  will  codify 
the  major  concepts  and  relations  that  underlie  high-quality  assessments  (e.g.,  Linn  et 
al.,  1991).  A  constraint-processing  engine  will  evaluate  the  fit  between  the  user's 
design  and  the  assessment  ontology  in  the  context  of  the  domain  ontology,  and  alert 
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the  user  of  constraint  violations  as  well  as  options  in  the  design  that  would  satisfy 
all  constraints  (Montanari,  1974). 

Assessment  Design  as  Constraint  Satisfaction 

For  the  purpose  of  assessment  design,  a  constraint  network  can  be  used  to 
explicitly  represent  an  assessment  model  and  provide  a  description  of  assessment 
parameters,  the  constraints  governing  relationships  among  the  parameters,  and 
computational  access  to  the  parameters  and  constraints.  This  representation  can  be 
used,  for  example,  to  provide  guidance  to  assessment  authors  as  they  design 
assessments  for  particular  purposes  under  particular  constraints.  That  is,  the  explicit 
structure  is  of  very  high  utility  because  it  allows  the  enforcement  of  a  common  and 
consistent  framework.  This  structure  can  be  leveraged  to  assist  assessment  authors 
(particularly  non-experts)  in  designing  assessments.  Assessment  authoring  support 
could  be  in  the  form  of  (a)  aiding  assessment  authors  to  populate  the  assessment 
ontology  with  values  specific  to  the  users'  purposes,  and  (b)  system  constraint 
checking  that  would  ensure  that  assessment  authors  are  alerted  to  incompatible 
values. 

The  specification  of  the  different  components  of  the  assessment  is  the  critical 
first  step.  We  have  defined  the  top-level  components  and  allowable  states  as  shown 
in  Table  1.  Table  2  shows  18  constraints  that  capture  the  model  described  in  Baker 
and  O'Neil  (2002).  These  constraints  specify  the  allowable  conditions  of  problem¬ 
solving  task  design.  For  each  constraint,  the  variable  and  value(s)  are  listed  in  the 
following  format:  <category>::<category  variable>::(possible  values).  A  constraint  is 
satisfied  if  all  variables  are  set  to  the  specified  values.  A  design  is  acceptable  if  all 
constraints  are  satisfied.  Note  that  the  model  in  Table  2  represents  one  point  of  view 
of  a  problem-solving  assessment  model.  Other  perspectives  will  have  different 
representations  and  constraints.  While  this  idea  is  not  new  (e.g.,  see  Baker,  1997), 
what  has  changed  is  the  availability  of  computational  tools  that  make  feasible  the 
capturing  and  processing  of  the  model  computationally. 


Knowledge,  Models,  and  Tools  to  Improve  the  Effectiveness  of  Naval  Distance  Learning  17 


Table  2 


Constraints  Associated  With  Designing  a  Problem-Solving  Task 


Set 

No. 

Constraints 

1 

COGNITIVE  DEMAND::-::(reasoning) 

PROBLEM  IDENTIFICATION::explicitness::(partially  identified,  embedded,  multiply-masked) 
PROBLEM  IDENTIFICATION::barriers  to  getting  information::(allow  for  barriers) 

PROBLEM  IDENTIFICATION: prior  knowledge::(low  prior  knowledge,  high  prior  knowledge) 

2 

COGNITIVE  DEMAND::-::(content  understanding) 

PROBLEM  IDENTIFICATION::explicitness::(partially  identified,  embedded,  multiply-masked) 

PROBLEM  IDENTIFICATION::barriers  to  getting  information::(none) 

PROBLEM  IDENTIFICATION: prior  knowledge::(high  prior  knowledge) 

3 

COGNITIVE  DEMAND: :-: : (factual  knowledge) 

PROBLEM  IDENTIFICATION::explicitness::(stated) 

PROBLEM  IDENTIFICATION: prior  knowledge::(high  prior  knowledge) 

4 

PROBLEM  IDENTIFICATION::explicitness::(stated,  embedded) 

PROBLEM  IDENTIFICATION: prior  knowledge::(low  prior  knowledge) 

5 

PROBLEM  IDENTIFICATION::explicitness::(stated,  embedded) 

SOLUTION  CHARACTERISTICS;:sub-solution  contingencies::(non-sequential) 

6 

PROBLEM  CFLARACTERIZATION::type  of  task::(improvise  steps,  combination  of  tasks) 
SOLUTION  STRATEGY::steps::(non-specified  course  of  action) 

7 

SOLUTION  STRATEGY: contingency  planning::(not  required) 

SOLUTION  CHARACTERISTICS: solution  space::(convergent) 

SOLUTION  CHARACTERISTICS::sub-solution  contingencies::(non-sequential) 

8 

SOLUTION  STRATEGY: contingency  planning::(required) 

SOLUTION  CHARACTERISTICS: :sub-solution  contingencies::(sequential) 

9 

PROBLEM  CHARACTERIZATION : : type  of  task::(execute,  fix) 

SOLUTION  STRATEGY::steps::(explicit  course  of  action) 

10 

PROBLEM  CHARACTERIZATION : :type  of  task::(execute,  fix) 

SOLUTION  CHARACTERISTICS::solution  space::(convergent) 

SOLUTION  CHARACTERISTICS::solution  correctness::(single  acceptable  solution) 

11 

PROBLEM  IDENTIFICATION: prior  knowledge::(low  prior  knowledge) 

SOLUTION  STRATEGY: cognitive  strategies::(domain  independent) 

12  PROBLEM  IDENTIFICATION: rprior  knowledge::(high  prior  knowledge) 
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Set 

Constraints 

No. 

PROBLEM  CHARACTERIZATION: :type  of  task::(execute) 

SOLUTION  STRATEGY::cognitive  strategies::(domain  dependent) 

13 

PROBLEM  IDENTIFICATION::time  constraints::(self-paced) 

SOLUTION  STRATEGY::cognitive  strategies::(domain  dependent) 

14 

PROBLEM  IDENTIFICATION::time  constraints::(extemaI) 

SOLUTION  STRATEGY::cognitive  strategies::(domain  independent) 

15 

PROBLEM  IDENTIFICATION::explicitness::(stated) 

PROBLEM  IDENTIFICATION: :number  of  information  sources::(zero) 

16 

PROBLEM  IDENTIFICATION::explicitness::(partially  identified,  embedded,  multiply-masked) 

PROBLEM  IDENTIFICATION::number  of  information  sources::(single,  multiple) 

17 

PROBLEM  IDENTIFICATION::explicitness::(stated,  partially  identified) 

SOLUTION  STRATEGY::problem  subdivision::(not  required) 

18 

PROBLEM  IDENTIFICATION::explicitness::(embedded,  multiply-masked) 

SOLUTION  STRATEGY: :problem  subdivision::(required) 

Applying  the  Framework:  Examples  From  Two  Domains 

In  an  ideal  assessment  design,  the  specifications  of  the  task  would  be 
determined  by  how  the  information  from  the  assessment  would  be  used.  The 
purpose  of  the  assessment  should  drive  the  complexity  of  the  problem  as  well  as 
define  what  sort  of  strategies  are  necessary  to  solve  the  problem.  The  following 
examples  demonstrate  this  notion  of  matching  the  task  specifications  with  the 
assessment  purpose  (the  appendix  contains  an  additional  example).  Using  the 
scenarios  mentioned  earlier  in  the  context  of  a  marksmanship  coach  course  which 
trains  Marines  on  how  to  be  marksmanship  coaches,  we  specify  the  assessment 
component  values  with  respect  to  the  goal  of  the  assessment. 

Example  1.  Determining  What  Students  Already  Know 

This  first  example  demonstrates  what  the  authoring  process  of  a  good 
assessment  would  look  like  for  Scenario  1,  "determining  what  students  already 
know."  This  is  from  the  view  of  an  instructor  administering  the  assessment  at  the 
beginning  of  a  course  trying  to  get  a  sense  of  how  much  knowledge  the  students 
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have  about  the  marksmanship  coaching  process.  The  task  specifications  of  this 
example  are  found  in  Table  3. 

A  new  course  is  starting  and  the  students  are  coming  with  a  wide  range  of  backgrounds. 

The  instructor  wants  to  get  a  general  measure  of  a  student's  knowledge  about 
marksmanship  and  the  marksmanship  coaching  process.  The  assessment  is  administered 
prior  to  any  instruction  in  order  to  make  adjustments  to  the  curriculum.  The  task 
administered  to  the  student  involves  identifying  a  shooter's  problem  by  looking  at 
various  information  sources  (e.g.,  data  book,  target,  shooter's  position,  weapon  settings, 
etc.)  and  prescribing  its  appropriate  solution  under  the  same  time  constraints  as  a  coach 
on  a  range  (approximately  5  minutes).  In  this  particular  example,  the  shooter's  problem 
is  with  windage  settings — the  shooter  had  not  properly  compensated  for  wind  prior  to 
firing.  Therefore,  all  of  the  shots  are  forming  a  group  to  the  left  of  the  center  of  the  target. 

Critique.  Very  little  guidance  on  identifying  or  solving  the  shooter's  problem  will  be 
provided  to  the  student.  Therefore,  performance  on  this  task  is  highly  dependent  upon 
the  student's  prior  knowledge,  with  students  who  know  more  about  marksmanship 
expected  to  do  better  than  those  who  do  not  know  as  much.  Because  the  instructor  is  not 
particularly  interested  in  whether  a  student  knows  what  to  do  if  there  is  difficulty  with 
an  information  source  (either  because  of  availability,  accuracy,  completeness,  or 
consistency)  all  of  the  sources  are  available  (including  those  that  may  be  irrelevant  to 
identifying  and  solving  the  problem),  accurate,  and  consistent  both  within  and  among 
each  other.  Once  the  shooter's  problem  has  been  identified,  the  assessment  requires  that 
the  student  "fix"  the  problem  (adjust  the  shooter's  windage).  The  remedy  is  fairly 
straightforward,  so  the  student  is  not  required  to  come  up  with  a  creative  solution. 

Example  2.  Determining  the  Degree  to  Which  Students  Have  Learned  Requisite 
Skills  and  Knowledge 

This  example  is  what  the  authoring  process  of  a  poorly  designed  assessment 
would  look  like  for  Scenario  3,  "determining  the  degree  to  which  the  students  have 
learned  requisite  skills  and  knowledge."  In  this  case,  the  instructor  administers  the 
assessment  task  at  the  end  of  the  course  to  see  whether  the  students  are  prepared  to 
begin  coaching  on  the  range.  The  task  specifications  of  this  example  are  found  in 
Table  3.  The  aspects  of  the  task  that  contribute  to  its  poor  design  are  italicized. 

The  coach  course  has  been  completed  and  the  instructor  is  unsure  if  several  of  the 
students  are  prepared  to  coach  on  the  range.  The  assessment's  purpose  is  to  see  in  which 
areas  the  students  are  still  having  difficulty.  The  instructor  is  particularly  concerned  that 
students  do  not  know  what  to  do  if  a  piece  of  information  is  missing  in  order  to  make  a 
diagnosis. 
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Critique.  This  design  has  two  key  flaws:  (a)  explicitness  of  problem  identification  and  (b) 
the  quality  of  information  sources.  Problem  identification  has  very  little  ambiguity  which 
does  not  allow  adequate  demonstration  of  knowledge.  The  task  guides  the  student  as  to 
where  they  should  look  to  figure  out  what  is  the  shooter's  problem  (instructor  chooses  a 
problem  statement  that  says,  "Shooter's  problem  has  something  to  do  with  her  weapon").  The 
instructor  is  also  looking  for  the  ability  to  deal  with  complications  such  as  missing 
information  because  coaches  often  encounter  this  situation  when  it  is  raining.  However, 
the  instructor  designed  a  task  that  had  no  barriers  to  getting  information,  and  completeness  of 
information  sources.  By  choosing  these  states,  the  assessment  will  make  everything 
available  to  the  student  and  the  instructor  will  not  be  able  to  measure  competence  in 
dealing  with  missing  information. 
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Consistency  within  info,  sources  1,  2  consistent  For  example,  the  wind  values  are  consistent.  From  the  time  before  firing  the  shots 

commence  to  the  end  of  the  stage,  the  wind  values  remain  the  same. 

Accuracy  of  info,  sources  1, 2  accurate  All  of  the  information  is  accurate. 

Completeness  of  info,  sources  1  complete  Everything  is  presented  in  the  scenario,  including  less  useful  information. 


Cognitive  strategies  1,  2  domain  Problem  needs  to  be  solved  using  specific  marksmanship  coaching  strategies. 

dependent 

SOLUTION  CHARACTERISTICS  1  convergent  The  problem  has  a  known  solution:  to  adjust  the  windage  settings. 

Solution  space 
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Example  3.  Mismatch  Between  Assessment  Purpose  and  Assessment  Design 

This  example  comes  from  an  assessment  used  in  a  mathematics  lesson 
(reported  in  Koency,  2000).  This  example  illustrates  how  the  intended  purpose  of  the 
assessment  can  be  undermined  by  the  interaction  between  prompt  and  information 
source.  In  this  case,  the  instructor  administered  the  task  at  the  end  of  the  unit  to 
gather  information  on  whether  students  had  mastered  fractions.  The  teacher's 
objectives  of  the  unit  were:  (a)  students  will  understand  percents  as  an  alternative 
way  of  representing  fractions  with  a  denominator  of  100  and  (b)  students  will  build 
an  understanding  of  the  relationships  between  fractions,  decimals,  and  percents.  The 
use  of  the  assessment  and  critique  are  described  next. 

At  the  end  of  the  unit,  the  teacher  gave  students  the  worksheet  shown  in  Figure  1.  The 
students  were  given  the  prompt  "Which  one  is  the  better  sale  (A  or  B)?"  and  were  asked 
to  provide  a  written  explanation  justifying  their  choice.  Students  were  required  to  record 
the  sale  price  for  each  piece  of  luggage  under  the  "Sale"  column.  Using  the  variables  in 
Table  1,  the  task  can  be  described  as  shown  in  Table  4.  The  major  aspects  of  the  task  are: 

(a)  assessment  purpose  ( certification ),  (b)  cognitive  demand  ( content  understanding),  (c) 
explicitness  of  problem  identification  (embedded),  and  (d)  relevancy  of  information 
sources  (allow  for  no  relevancy  of  information  sources). 

Critique.  The  teacher's  purpose  was  to  assess  whether  the  students  understood  the 
meaning  of  percentages  (certification  and  content  understanding)  by  using  percentage 
arithmetic  to  calculate  the  savings  and  make  a  quantity  comparison  between  two  values. 
Students  were  asked  to  report  the  various  strategies  they  employed  to  find  the  new 
luggage  prices.  However,  the  teacher  designed  a  task  that  had  embedded  problem 
identification  that  contained  information  sources  that  allowed  for  no  relevancy  types.  The 
prompt  in  conjunction  with  the  information  on  the  worksheet  created  a  potentially 
ambiguous  situation.  The  problem  asked  students  to  determine  "which  one  is  the  better 
sale"  but  the  assessment  scenario  contained  a  number  of  irrelevant  information  sources 
(e.g.,  warranty  information,  luggage  features,  wheels)  that  were  not  needed  to  solve  the 
problem.  An  embedded  problem  is  appropriate  for  tasks  requiring  the  cognitive  demand 
of  reasoning. 

The  ambiguity  lay  with  the  potential  for  a  student  to  interpret  the  task  as  one  of 
considering  the  additional  information  (e.g.,  warranty  and  luggage  features)  with  cost — a 
very  reasonable  approach,  especially  given  the  "authentic"  context  of  the  task.  Poor 
performance  on  the  task  may  be  due  to  a  misinterpreting  what  the  problem  is  asking  for 
rather  than  not  knowing  the  material.  This  source  of  construct-irrelevant  variance 
subverts  the  intended  purpose  of  the  assessment,  to  provide  information  on  whether 
students  have  attained  the  objectives. 
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A.  Our  Most  Exclusive  Brand- 


lightly  woven  softs*!*  luggage  in  bright  red  or  green.  Larger 
pieces  on  wheel*  for  easy  maneuverability  1 0-year  warranty. 


A.  Wheeled  garment  bag  $228 

8.  Attachable  multi-pocket  tote  $94 

C.  2S*  upright  pullman  $215 

D.  Multi-pocket  cany-on  $80 

E.  26"  upright  pullman  $215 

F.  22"  upright  suiter  $175 


Reg.  Sale 
$228 


m 


h  v*r 


B.  Famous  Maker  Luggage— 50%  of 


Durable  and  lightweight  made  of  1200-deruer  polyester. 
Sturdy  black  rubber  wheels.  5-year  warranty. 


M. 

i 


G.  30*  upright 

H.  26*  upright 

I.  Garment  bag  wf  wheals 

J.  32”  trolley  duffel  bag 

K.  Multi-pocket  carry-on 


Reg 

$260 

$200 

$260 

$200 

$70 


s  r, 

Ml 


Which  one  is  the  better  sale?  (AorB).  Write  your  explanation  on  the  back 
of  this  page. 


Figure  1.  Example  3  information  source. 


26 


Center  for  the  Study  of  Evaluation 


Example  4.  Correcting  the  Mismatch  Between  Teacher's  Assessment  Purpose  and 
Assessment  Design  in  Example  3 

The  problem  in  Example  3  lies  in  the  four  incompatible  values  of  the 
assessment  design:  certification  (purpose),  cognitive  demand  (content  understanding), 
explicitness  (embedded),  and  relevancy  of  information  sources  (allow  for  no  relevancy  of 
information  source).  To  fix  the  task  design  in  Example  3,  the  explicitness  of  the 
problem  to  be  identified  should  be  stated.  An  example  of  an  explicit  prompt  is: 
"Based  on  the  calculated  savings,  determine  which  offer  is  the  better  sale."  The 
prompt  makes  explicit  that  students  should  base  their  decision  only  on  the 
calculated  savings.  Presumably,  students  will  apply  only  their  knowledge  of 
percentages  (content  understanding)  to  carry  out  the  task,  rather  than  considering  the 
extraneous  information  contained  in  the  scenario  (reasoning).  The  slight  adjustment 
to  the  prompt  aligns  the  assessment  purpose  with  cognitive  demand  (content 
understanding)  and  problem  identification  explicitness  (stated).  The  revised  task 
specifications  are  shown  as  Example  4  in  Table  4. 
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between  a  student's  performance  and  the  student's  actual  knozvledge. 
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Discussion  and  Next  Steps 

In  this  report  we  outlined  an  approach  to  the  design  of  an  authoring  system  for 
problem-solving  tasks.  A  key  feature  to  support  evaluation  of  assessment  designs  is 
the  use  of  a  constraint  network  to  capture  the  allowable  relations  among  the 
assessment  model  variables.  The  basic  idea  is  that  major  assessment  and  task 
variables  can  assume  a  fixed  set  of  values,  and  that  constraints  among  the  variables 
define  allowable  relations  or  conditions.  A  task  design  can  then  be  validated  by 
checking  for  constraint  violations. 

Technologies  that  can  support  the  assessment  authoring  design  process  seem 
particularly  promising  because  of  the  nature  of  the  anticipated  users:  non-experts 
who  lack  breadth  and  depth  of  knowledge  of  assessment.  A  constraint-based 
authoring  system  can  impose  structure  on  the  authoring  process,  focusing  users' 
attention  on  the  important  variables  underlying  the  assessment  task,  as  well  as 
verifying  that  user-specified  values  are  consistent  with  the  underlying  assessment 
model.  The  advantage  of  a  constraint-based  approach  is  a  tighter  coupling  between 
the  assessment  model  and  the  instantiated  task.  Presumably,  the  more  the  task 
design  adheres  to  the  assessment  model,  the  higher  the  task  quality — this  would 
make  the  authoring  system  particularly  useful  for  novice  assessment  authors. 

While  we  are  confident  that  the  technology  component  (i.e.,  constraint 
processing)  exists,  the  larger  assessment  issue  is  whether  the  domain  can  be 
captured  in  terms  of  variables  and  states,  and  whether  such  technology-enabled 
solutions  result  in  higher  quality  assessments  (Baker,  2003).  We  have  attempted  to 
illustrate  how  problem-solving  tasks  could  be  represented  with  constraints  and  how 
constraints  could  "catch"  design  flaws,  but  more  work  is  needed  to  test  the  notion  of 
constraints  applied  to  assessment  design,  the  variety  of  constraints,  their 
interactions,  and  the  implications  for  designing  tasks. 

Next  steps  for  this  work  include  gathering  evidence  on  the  degree  to  which  our 
framework  yields  judgments  similar  to  those  of  experts  for  good  and  poor 
assessments.  Constraint  violations  should  be  detected  for  poor  tasks  and  be  absent 
for  good  assessments.  Such  evidence  would  support  the  interpretation  that  the 
framework  was  capturing  meaningful  aspects  of  the  assessment  design.  A  second 
test  is  to  examine  how  the  authoring  system  would  work  for  authors  of  differing 
backgrounds.  For  example,  one  set  of  comparisons  is  between  novice  test  designers 
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with  and  without  the  authoring  framework  (e.g.,  typical  classroom  instructors). 
Effectiveness  of  our  system  would  be  evidenced  by  higher  quality  assessments 
created  by  novices  using  our  system  compared  to  those  not  using  our  system. 

As  assessment  design  moves  from  craft  knowledge  to  an  engineering 
discipline,  models  and  tools  will  be  needed  to  facilitate  the  systematic  development 
of  assessments.  We  have  presented  one  approach  that  could  be  implemented  using 
constraint  networks  to  support  the  assessment  design  process. 
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Appendix 
Worked  Examples 
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identify  the  problem. 

Content  understanding  Student  is  told  that  the  shooter's  problem  may  be  with  the  Student  is  given  the  baby's  symptoms  (fever,  change  in 

shooter's  positioning.  eating  habits,  diarrhea,  and  fussiness). 

The  student  needs  to  demonstrate  understanding  of  The  student  needs  to  demonstrate  understanding  of 

shooting  positions  to  solve  the  problem.  how  to  make  a  diagnosis  given  a  list  of  symptoms. 


3 


12 

£ 

U  -K 

o  Mh 

•is  cu 


15  cn  rg  .5 

£  'g  £  1 

2  c  4)  ® 

■*t  c  ^  o 

<P  GJ  4-> 

2  "2  >  o- 

£  5  o  £ 

|  ;  § 

I  £  o  ,5  £ 


1  C/5 

SC/5 
C/5 

o  o» 

ex -5 
1,1 

>>  2 
Cft  ^ 

eft  t: 

I  V  c 

.o  « 

5  « 
£>  0) 
b  -C 

Xv- 
v- 

"2  n 
S  "3 

2  Cft 

:  t>o  .-t: 

eft  X> 

— 1  m 

4-  jC 

C  V ' 

O)  oJD 

-  .5 

2  15 

if)  o> 


<U  (ft 

£  2  g? 

"  M-£ 

■g  .£  b 

2  >  « 

gif 

en  ^r>.£ 

•"  a  £ 

£  g  -w 

b  .£  x, 

S  <"  .SP 

O  s  <« 

b  vh  >- 
CL  •£  a; 

^  §-  2 
SSii 
o  .2 

O  c/5  ^ 

JZ  «  D- 
cn  H  O 
b  £ 

£  "O  jx 

2  2  « 

_g  '3  S 

>  o-t; 

■a  2 

fils® 

.2  M  ~T  £ 
.2  £  c 
S  Ms  b  £ 

^  -p  r;  in 

■o  e  -8  3 
2  .2  2  ^ 
cn  .Js  cl  as 


0> 

E 

M-t 

o 

<P 

1 

Vh 

2 

c 

<P 

u 

J-l 

X 

o 

Oh 

4-* 

0 

CD 

05 

o 

T5 

Vl 

X 

C/5 

0) 

jp 

O 

o 

JO 

<P 

4—' 

0> 

C 

4— ' 

O 

X 

rr\ 

o 

4— > 

bO 

O 

05 

X 

m  jr 

O  £ 

<4-<  ^ 

Hi 
I  •? 


:  2  o 

b 

■3  « 

i  2  2 

e/)  s 


| j  g  O  E 

b  ts  4--  o 

~o  >  c  'S 

2  %  §  -S 

(«  b  ■-  b 
S  b  o  oj 

P  & 


(8  b 

>  60 

Zi  tj 

T}  ai 


■C  £ 

*  IS 

‘C  to 


pa  b 

§  ^ 

K  x 

PH  W  : 


ON 

CO 


4) 

3 

to 

> 

§ 

0) 

1 

i 

> 


c/5  CD 


X5 

CD 

'd 

0) 

I 

w 


X 

to 

o 

o 

£ 


to 

£ 

T3 

'o 


s 

T3 

B 

c_n 


<D 

o 

,o 

of 


(0 

60 


.el 

X 

to 

>  .6 
CO  ■H 

c 

o 

X 


to 

41 


6^ 

X) 

10 

X 


c 

£  S 

UN 

(0 


41 

u 

e 

(6 

X 

4) 

cn 

2 


4J 

)H 

5 

<8 

>-i  2 

4)  X 

C 

g  a 

6  §< 

4) 

4)  cn 

U 

Oh  £ 
c/5  OJ 


C/5 


"o 

cLS! 

4)  ~ 

£ 


c 

4> 

g 

4J 


o 

g 


j-  4) 

■  5 


-o 

41 

cn 

(0 

g 


Oh 

3 

S 


to  .a 


g 

x 

to 

g 

I 

M 


2  c 
a-1  .2 

•rl  T3 

fcl  a 

to  ■*? 


41 

g 

2 


03 

3 

x 

u 

1 

o 

CD 


bo 

6 

.3 

c 

u 

o; 

IS 

*£h 

13 

ir 

cn 

0> 

X 

£ 

CD 

u 

O 

4~* 

o 

> 

o 

X 

g 

4-* 

13 

S 

£ 

4— » 

u 

4-» 

o 

0) 

Jh 

3 

.5 


03 

Oh 

4-* 

03 

1 


cd 

bJD 


I 

g 

T3 

CD 

3 

u 

C/3 

•8 

a 

os 

C/5 

1 

-M 

CD 


’g 

to 

o 

u 

4? 

£> 

JS 

4J 

3 

g 

TJ 

g 

a 

u 

cn 

-§ 


'£ 

CO 


§ 

a  © 

CD  CD 


C/5 

tH 

CD 

*0 

Vh 

03 

X 

o 


o 

X 

C/5 

1 

a 

! 

© 

© 

cn 

O 

o 

o 

JD 

2 
tO 

t3 

© 

I 

I 

X 

CD 

o 

4-» 

T3 

CD 

§ 


to  rely  on  nonverbal  cues — crying,  kicking,  etc.  The 
temperature  is  important — if  baby's  movement  makes 
it  impossible  to  take,  student  may  have  to  determine 
whether  he  has  a  fever  by  touching  his  forehead. 
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Allow  inconsistencies  Wind  values  are  inconsistent.  Temperature  readings  are  inconsistent. 

The  weather  is  changing  over  the  course  of  fire.  Sometimes  the  temperature  is  101°  and  other  times  it  is 

98.7°. 
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All  of  the  information  sources  are  missing — there  is  no  data  The  parent  cannot  tell  that  the  baby  has  a  sore  throat  as 
book,  target,  range,  or  weapon.  well  because  the  baby  cannot  talk.  This  is  an  essential 

piece  of  information  that  cannot  be  inferred  from  an 
available  source. 
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8  '  Student  knows  the  source  of  the  baby's  sickness,  so 

Student  knows  that  the  source  of  the  shooter's  problem  is  there  is  no  need  to  break  it  down  further, 
her  windage  settings.  There  is  no  need  to  break  it  down 
further. 
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algorithms.  Parent  uses  trial  and  error  to  see  if  he  can  identify 

the  baby  is  sick. 

Domain  dependent  Problem  needs  to  be  solved  using  specific  marksmanship  Problem  needs  to  be  solved  using  specific  medical 

coaching  strategies.  techniques. 
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